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Abstract 

We present in this work a new family of kernels to compare positive 
measures on arbitrary spaces X endowed with a positive kernel k, which 
translates naturally into kernels between histograms or clouds of points. 
We first cover the case where X is Euclidian, and focus on kernels which 
take into account the variance matrix of the mixture of two measures to 
compute their similarity. The kernels we define are semigroup kernels in 
the sense that they only use the sum of two measures to compare them, 
and spectral in the sense that they only use the eigenspectrum of the 
variance matrix of this mixture. We show that such a family of kernels 
has close bonds with the laplace transforms of nonnegative-valued func- 
tions defined on the cone of positive semidefinite matrices, and we present 
some closed formulas that can be derived as special cases of such inte- 
gral expressions. By focusing further on functions which are invariant 
to the addition of a null eigenvalue to the spectrum of the variance ma- 
trix, we can define kernels between atomic measures on arbitrary spaces 
X endowed with a kernel k by using directly the eigenvalues of the cen- 
tered Gram matrix of the joined support of the compared measures. We 
provide explicit formulas suited for applications and present preliminary 
experiments to illustrate the interest of the approach. 



1 Introduction 



Defining meaningful kernels on positive measures is an important issue in the 
field of kernel methods, as it encompasses the topic of comparing histograms, 
bags-of-components or clouds of points, which all arise very frequently in appli- 
cations dealing with structured data. 

In the pioneering applications of support vector machines to structured data, 
histograms were often treated as simple v ectors and used as such through the 
standard Gaussian or polynomial kernels [joa02l|. Yet, more adequate kernels 
which exploit the specificities of histograms have been proposed since. Namely, 
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the fact that histograms are vectors with nonnegative coordinates [HBOS] , and 
whose sum may be normahzed to one, that is cast as discr ete pr obabihty mea- 
sures and treated under the light of information geometry [LL05llLcb06]'. bmce 
such histograms are usuaUy defined on bins which are not equally dissimilar, as is 
for instance the case with color, words or amino-acid histograms, further kernels 
which ma y take i nto account an a priori inter-bin similarity where subsequently 
proposed [KJOl ICFVOSl . iHBOSj as an attempt to include with more accuracy 



a prior knowledge on the considered components, through the knowledge of a 
prior kernel k for instance. 

In this paper we investigate further such kind of kernels between two mea- 
sures, which can conveniently describe the similarity between two clouds of 
points by only considerin g their G ram matrices. In this sense we reformulate 
and extend the results of CFVOSj whose framework we briefly recall: 



The set M^(A') of bounded positive measures on a set A" is a cone, and 
from a more elementary algebraic viewpoint a semigroup0. In that sense, a 
natural way to define kernels suited to the geometry of M^(X) is to study the 
family of semigroup functions on M^(A'), as introduced in |BCR84 |. that is real- 



valued functions defined on M^(A') such that the map (/z, fj,') i-^ tpilJ- + m') is 
either positive or negative definite. The Jensen-divergence, which is computed 
through th e entr opy of the mixture of two measures is such an example, as 



recalled in [HB05| 



Given t he complexity of evaluating entropies for finite samples, it is shown 

in 



CFVOSj l that similar quantities can be defined for measures by only taking 
into account the variance matrix + of the mixture of two measures. This 
has two clear advantages. First, variances are easy to compute given atomic 
measures, that is measures with finite support, which are usually considered 
in most applications. Second, the eigenspectrum of the variance matrix of an 
(atomic) probability measure is known to be the same as, up to zero eigenvalues 
and an adequate centralization, the eigenspectrum of the dot-product matrix of 
the support of the same measure. This fact paves the way to consider kernels 
defined on Gram matrices rather tha n on v ariance matrices, regardless of the 



structure of X, as was first hinted in IKJ03 I. 

More precisely, the authors of [CFVOSj first prove that for a variance matrix 
-I- ^'), the determinant |-S(/z + /z') -t- /„|^2 for > is a positive definite 
(p.d.) kernel between two measures fi, /z' on an Euclidian space X of dimension 
n. Second, they prove that this quantity can be cast into a reproducing kernel 
Hilbert space (rkHs) associated with a kernel k on X, regardless of the nature 
of X, by using directly a centered Gram matrix /C^.^' of all elements contained 
in the support of both /i and fj,'. 

We are interested in this paper in characterizing other functions f defined on 
matrices such that (i) /i, fj,' (/j(S(/i-|-/i')) is either positive or negative definite, 
(a) (f is spectraH and (Hi) is invariant to the addition of null eigenvalues. 



^In this paper, a semigroup will be a non-empty set S endowed with a commutative addi- 
tion, such that for s,t £ S, their sum s + t = t + s(^S, and a neutral element e such that 



s + e 
2 



A function / defined on symmetric matrices is spectral, or orthogonally invariant, if for 
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that is, for two square p.d. matrices A, B which may not have the same size, 
(p{A) = ip{B) if A and B have the same positive eigenvalues taken with their 
multipHcity, regardless of the multiplicity of in their eigenspectrum. It is easy 
to check that both \ - ■ +I\ and the trace fulfill condition (Hi). 

If if satisfies condition (i) and (ii), we call the composed function ip = ipoT, 
a semigroup spectral positive (resp. negative) definite (s.s.p.d., resp. s.s.n.d.) 
function on AI^{X). Note that the task of defining such functions -0 is not 
equivalent to defining directly positive or negative definite functions on the 
semigroup of p.d. matrices, since the underlying semigroup operation is the 
addition of measures and not that of the variance matrices of the measures, as 
recalled in Equation ([!]). When ip is further invariant to null eigenvalues (Hi) 
tp can be cast in Hilbert spaces of infinite dimensions to compare degenerated 
variance operators, which will be in the context of this paper an rkHs built on 
X through a kernel k. 

This paper is structured as follows: we introduce in Section [2] an allevi- 
ated formalism for semigroup functions, and propose a general link between 
s.s.p.d. functions and the Laplace transform of functions defined on matrices 
in Section [3l We review then in Section |4] different s.s.p.d. functions, notably 
a function which satisfies criteria (Hi) and which does not requite any regu- 
larization. We provide explicit formulas and test the kernel derived from such 
a function on a benchmark classification task involving handwritten digits in 
Section El 

2 Semigroup Functions on Bounded Subsets of 

We consider X, an Euclidian space of dimension n endowed with Lebesgue's 
measure and restrict M^{X) to measures with finite first and second moments. 
In such a case, the variance of a measure /i of M^{X) can be defined as: 

Writing fl for we recall an elementary result for two measures of 

M|(A'), 

+ ti') = + S(m') - + p'fl^) , (1) 

which highlights the nonlinearity of the variance mapping. 

We write P„ for the cone of real, symmetric and positive semidefinite ma- 
trices, and P+ for its subset of (strictly) p.d. matrices. In this paper, the 
assumption that for a measure fi its variance T,{fi) is in P„ is crucial for most 
calculations, and this is ensured for sub-probability measures, that is is measures 
II such that = IJ.{X) < 1, since we then have that 

- ti[ix ~fl)ix- fif] + (1 - fipJ e P„. (2) 

any real n x n orthogonal matrix H, that is such that HHj_= I„, f(HAH^) = f{A). In 
that case / only depends on the eigenspectrum of A. See [BLOd l 
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Furthermore, we will also need the identity = fi[{x — p) {x ~ jx) ] in order 
to make the link between the dot-product matrix of the support of and its 
variance matrix, which is why we restrict our study to probability measures 
M\{X). M]^{X) is not, however, a semigroup, since it is not closed under 
addition, due to the constraint on \ix\. To cope with this contradiction, that is 
to use semigroup-like functions of the type (//, n') ip^jj. + n') where ip is only 
defined on a subset of the original semigroup, and where this subset may not be 
itself a semigroup, we define the following extension to the original definition of 
semigroup functions which, although technical, is also useful to recall the actual 
definitions of positive and negative definiteness for semigroup functions. 

Definition 1 (Semigroup kernels on subsets) Let (S, +) be a semigroup 
and U <Z S a nonempty subset of S . A function ip : U W is a p.d. (resp. n.d.) 
semigroup function on U if 

CiCj'ipisi + Sj) > {resp < 0) 

holds for any n € N; any si, . . . , s„ G 5 such that Si + Sj £ U for 1 < i < j < n; 
and any ci . . . , c„ € R (resp. with the additional condition that Ci — 0) 

In practice, stating that a function ip defined on the subset M^{X) is positive 
(resp. negative) definite is equivalent to stating that the kernel for two elements 
H, fi' of M]_{X) defined as 

is positive (resp. negative) definite. Finally, we write I]^^(/i) for {^{^))^^ when 
appropriate. 

3 Laplace Transforms of Matrix Functions and 
s.s.p.d. functions 

We show in this section how s.s.p.d. functions on M^{X) can be defined through 
the Laplace transform of a nonnegative- valued function defined on the cone P^, 
through the following lemma. 

Lemma 2 For any S G P„, the real-valued function defined on M^{X), 
is a negative definite semigroup function. 

Proof. For any fc G N, any ci,...,Cfe G R such that '^^Ci — and any 
Hi, . . . ,fik G M^{X) such that + fij G AI^{X), we have using Equation H]) 
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that 



\ I i,j 

Note that this function is actually n.d. for all measures of M^{X), regardless 

of their total weight \^\. The case S = In yields the simple function ipt^ i— s- 
trE(/x), which provides interesting results in practice, and boils down to a fast 
kernel on clouds of points, which we will review briefly in Section [5l For any 
nonnegative-valued function / : P+ R+ defined on the cone of p.d. matrices, 
we write 

Cf{Z) = / e-<^^'>f{S)dS (3) 
J sePt 

for the Laplace transform of / evaluated in Z E P+, when the integral exists. 
Proposition 3 For any spectral function f : P+ M+, the mapping 

defined for all measures pi G M]^{X) such that G P+ is a s.s.p.d. function. 

Proof. The integral w hen it exi sts is a sum of p.d. semigroup functions through 



Schoenberg's theorem [BCR84 Theorem 3.2.2], and is hence p.d.l 



Laplace trai isforms of functions defined on matrices is an extensive subject 



and we refer to jMat93l . Section 4] for a short survey. In the case where / = 1 we 
recover the characteristic function of the cone P+, and its logarithm, \n£l{A) = 
C — log is known as the universal barrier [Giil96( of the cone P+, with 
numerous applications in convex optimization. 

We recall now a well-known res ult of multivariate analysis based on zonal 



polynomials (see |Tak84 iMPHQSt for an exhaustive presentation of these), 



which may not, however, be of immediate use for an application in kernel meth- 
ods. To be short, zonal polynomials Cr,(A ) are pol ynomials in the eigenval- 
ues of a matrix A with positive coefficients |MPH95 . Remark 4.3.6], and thus 



nonnegative-valued spectral functions, indexed by the partitions a of an integer 
a. Namely, for a e N, we write a — (ai, 02, . . . , a„) for a partition of a into not 
more than n parts, that is ai + a2_+_- • • -I- a„ = a and ai > 02 > • • • > a„. The 



following result follows from [MPHQSl . Theorem 4.4.1] where we have dropped 



constants which only depend of n and a for more readability: 

Corollary 4 Given a G N and a partition a of a, the real-valued zonal kernel 
ipa is a s.s.p.d. function on M^{X), with 

Wa ■ fJ-^ T , 
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through the identity /5gp+ e-<^'^> |S'|*-^("+i)C„(S')dS' cx |E|-*C„(E-i), for 
t>^{n-l). 

Actual expressions for zonal polynomials of order a < 10 are currently known, 
and the use of Wishart densities for / can be seen as a special case of such 
evaluations. It is also clear that finite and infinite linear combinations of such 
zonal kernels, with the speculation that they might be a useful basis for a 
subcategory of s.s.p. d. functions, can be carried out in the spirit of equations 
provided in MPH95L Lemmas 4.4.5&6] and yield convenient formulas, such as 



,trS-MM)(ti.E-i(^))* 



a—k a 



e 



which is a s.s.p.d. function for any fc > 0. However, the weak point of these ex- 
pressions when used in our setting is that they tend to be extremely degenerated 
when the eigenspectrum of S vanishes, due to the high power of the denominator 
and to the fact that the eigenspectrum of not E, is considered implicitly. 
Hence, we do not see at the moment how one would obtain expressions sat- 
isfying condition (Hi), even through the use of regularization. To handle this 
problem, we focus in the next section on degenerated integrations, that is we 
consider an extension of the Laplace transform setting defined in Equation ([3]) 
to degenerated functions / defined on families of semidefinite matrices of P„ . 

4 Degenerated Integrations on Semidefinite Ma- 
trices of Rank 1 

We restrict the integration domain to only consider the subspace of P„ of ma- 
trices of rank 1, that is matrices of the form yy^ where y S K". The Euclidian 
norm y^y of y is the only positive eigenvalue of yy^ when y ^ Q, hence only 
real-valued functions of y^y can be spectral. Following the proof of Proposi- 
tion [3l and for any nonnegative- valued function g : M"*" M+, we observe thus 
that 

V'g : M ^ / e-y'^^^^yg{y'^y)dy (4) 



is a s.s.p.d. function on AI^{X), noting simply that tr{'S{fi)yy^) — y^'S{fi)y. 
We start our analysis with a simple example for g, which can be computed in 
close form. 

4.1 The case g : x ^-^ 

For a matrix A G such that mspec^ = {Ai, . . . , A„}, we set joiA) — 1 and 
write for 1 < i < n, 

dof riLiTOfc + 5) 



Aj^ • • • An" 



6 



where the summation is taken over all families ^ € N" such that the sum of their 
elements |j| is equal to i. Writing (T„ for (27r~2 )-2 we have with these notations 
that for all 1 < i < n, 

Corollary 5 The function tpi: On^Jy^-^i is a s.s.p.d. functions on 

MliX). 

Proof. Let /i £ M^{X), and write mspecE(/i) — {Ai,...,A„}. Then by an 
appropriate base change we have for gi : x t—i- , i < n, 



I -I - J 1 -^M I -I - J 1 



The inver se gene ralized variance is recovered as "00- We refer now to Lancaster's 
formulas BerOSl p. 320] to express more explicitly the cases i — 1,2, 3, where we 



write E for 



\/ E \/ E 



lJ;3{^l) [(trE-i)3 + 6(trE-i)(trS-2)+8trE 



-31 



Although the functions ipi are s.s.p.d., they are mainly defined by the lowest 
eigenvalues of E(/i). These functions can all be regularized, by adding a weighted 
identity matrix /„ to S, while still preserving their positive definiteness as can 
be easily justified by using the functions gi{x) = e~^x^ to penalize for large 
values of y. In such a case however, and to the notable exception of V'Oi this 
regularization prevents the above functions to be invariant to the addition of a 
zero eigenvalue to the spectrum of Intuitively, this degeneracy is due to 

the fact that we integrate on the whole or i?", notably on kerE(/i), where the 
contribution of exp(— y^I](/i)2/) is infinite. We propose to solve this issue by 
considering more specifically the contribution of each sphere {y\y^ y = t} to the 
overall summation in the case where g = 1. 

4.2 The case g : x t-^ 5t and its variants 

The question of integrating exp(— j/^Ej/) over compact balls {y G \ y^y < 
t} or spheres {y € M" | y^y = t} is closely related to the evaluation of the 
distribution of quadratic forms in normal variates MP92| . Given a matrix 



Q G Pn and a random vector y in ffi." following a normal law Af{m, V) with 
V € P+, the density h[Q, V, m] of the values of y^Qy, that is 



h[Q,V,m]{t)dt^ (271)-'! \V\-^ e-te-™)^v'-i(y-m)^j^^ 

Jt<ij^Qij<t+dt 
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can be evalua ted exp licitly in terms of the eigenvalues of V^QV^ through 



series expansions MP92{ , as well as the distribution of h and its Laplace trans- 



form. We note that these expressions are similar to that of the elementary 
contribution of the sphere {y^y — t} when 5 = 1, 

f,it)dt^ [ e-^"^('')^ dy. (6) 

Jt<y~''y<t+dt 

The difference between expressions © and ^ is that the variance E~^(yLt) 
may be potentially infinite if Equation Q is directly translated in terms of 
Equation ([5]), while the normalization term in does not appear in the 

s.s.p.d. function of Equation ([6]). It turns out that these two problems can 
be easily canceled out. We consider the Laplace transform of ffj.{t) defined as 
Lfj-i^) — /t>o ^ which is trivially a s.s.p.d. as a sum of elementary 

s.s.p.d. functions. 

Lemma 6 For mspec = {Ai, . . . , A„}, define the sequences dk — \ '^j 
for fc > 1, Co = 1 and Ck — j: X]r=o '^k-rCr for k > 1. Then, 

Proof. Take the exact value computation for 
ttt , ,1. 



\^{^l) + 2r^In\^ '^2 

and let 77 ^ to obtain through |MP92l Lemma 4.2b. 2 & Theorem 4.2b. 1] the 
corresponding formulas, both for and L^M 

Proposition 7 By considering s = \ in i^(s) from Lemma\^ and noting that 
dk = 5 tr(S(/i)'^') we have that 

oc 

is a s.s.p.d. function on M^{X), invariant to the addition of null eigenvalues, 
defined when the spectrum ofT,{fi) is strictly upper-bounded by 1. 

Although all terms dk are invariant by the addition of a null eigenvalue, 
cannot fulfill condition (Hi) because of the numerator in T{j + k) which depends 
on both the dimension n and the summation variable k. 

We refer to the proof [CFVOSl . Theorem 7] to show that for an atomic mea- 
sure fj, such that 1^1 — 1, that is ^ € Mol^(A'), Vm(m) can be either expressed 
in terms of the spectrum of its variance matrix or in the spectrum of its cen- 
tered dot-product matrix. Thus, if X is now an arbitrary space endowed with 
a kernel k, the centered Gram matrix corresponding to the support of /i can be 
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used directly as an input for a s.s.p.d. function such as ipM in order to define 
a p.d. kernel on Mol^(A'). We detail in Section [5] the formulation of such a 
kernel, along with two other examples. 

Finally, as a closing remark for this section, note that the representation 
proposed in Proposition [3] is not exhaustive to our knowledge, and should not 
be confused with the integral represent ation of semigroup p.d. functions as sums 
of semicharacters, studied in CFV05 |. First, the functions /i i— > e"^^*^^^'"^^ are 



not semicharacter^ of the semigroup (X) , since 

g-{S(/.+M'),S'> ^ g-(S(M)+S(M'),S> 

in the general case. Second, the class of functions considered through Lemmaj^jis 

far from characterizing all semigroup n.d. functions on M^{X) since, through BCR84 
Corrolary 3.2.10, p. 78], we have that for any 5 G P„ and < (3 < 1 both 

^I ^ 5)^ and ^l ^ ln(l + 5)), 

are semigroup n.d. functions. Note that if we use for y e M" and m > 1 a 
n.d. function of the type 

m + ri , 1 -Y^ 



— ^ — ln|^l + — y E(/i)y 

and exponentiate it in the spirit of Equation Q , we recover the integration of 
the Student multivariate distribution for vectors of R", which boils down again 
to a kernel that is proportional to tpQ. 



5 Explicit Formulas for Atomic Measures and 
Experiments 

Given two clouds of weighted points 7 = {xi, aj)f=i and 7' = (y^, bi)f^i, we show 
how to compute three different kernels which satisfy condition (Hi), namely tptr, 
V'o and i/'M, and compare them by studying their performance on a multiclass 
classification task. 



5.1 Formulations for Clouds of Points 

The mixture of 7 and 7' can be expressed as 7" = {{xi, ^ai)^^^, {yj, 56^)^^^}. 
By writing d" — d + d' and 

K.^ = [K{xi,Xj)]i^j<d, JC^ = [K{y^,yj)]i^j<d',a.iid /C, = [K{xi,yj)]i<d.j<d' 
we can further express the d" x d" Gram matrix of the mixture 7" as 

•^Semicharacters of a semigroup S are, following the definition of [BCR84| . real- valued 
functions p on 5 such that for s,t £ 5, p{s + t) = p{s)p(t) 
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As can be seen in CFV05| , the spectrum of the Gram matrices cannot be taken 



as such since they require a centrahzation of the form 

t-,n = (V - td'',d"Ay')K,y'{Id" - A^,,lrf.,,d")A^", (7) 

with A-y" = diag(iai, \hj) and Id",^" is the d" x d" matrix of ones. The exphcit 
formulas for the considered kernels, which we do not normalize and propose with 
bandwidth parameters are thus 

Ao(7,7') = l^^7"+Vr^ 

oo -j^ k — 1 ^ 

kmil, 7') = '^{-'^^Ck , with cq = 1, Cfe = - ^ dk-rCr and dk ^ - tr([(S^^//]'') 

fc=0 r=0 

with t > 0, 77 > and ^ > such that 5 < 1/p{ICy'), where for a matrix A G Pn 
such that mspec^ = {Ai,...,A„}, p{A) is the spectral radius of A, that is 
maxi<i<„ Xi. 

We discuss now possible values for S which will ensure that S < l/p{IC^) 
for any cloud-of-points 7 and any kernel k upper-bounded by one, that is 
supxex\iiix, x)\ < 1. Through Equation ([7]), one can obtain that for any cloud 
of points 7 = {xi, ai)f^i, 

p{jC^) < [max((i • a^ax - 1, 1)]^^ • a^ax 

where we write amax for the maximal weight of 7 and we have bounded p(/C-y) 
by d, which corresponds to the case )Cj = td.d- Thus, any factor d chosen so 
that 

^ ^ 1 

[max((i ■ to — 1, • lo 
can be used to compare families of clouds of points whose maximal weights do 
not exceed cu and maximal size does not exceed ^d. In the case where these 
clouds are bounded between dmin points (with weight l/dmin) and dmax points, 
this condition is ensured for d < (^™-)'^, which is far from being optimal in 
practical cases since the values of k are more likely to be better distributed in 
the [0,1] range. This shows however that if we compare clouds of similar size S 
can be equal to 1, and possibly above depending on the kernel k which is used. 
We leave for future work the study of the convergence of the series J2'k=a(~^)'''^>' 
corresponding to the evaluation of Ajm, although we note that in the practice 
of our experiments very few iterations (that is TV set between 10 and 20) are 
sufficient to converge to the limit value, which reduces considerably the overall 
computation cost with respect to a straightforward eigenvalue decomposition of 
IC-y" . Indeed, as is the case with the inverse generalized variance, this would 
have a cost of the order of while N computations of the traces tr[(5^-y"]'^ only 
grow in complexity Nd"^. It would be wise, however, to let N depend adaptively 
on the convergence of tr([(5AC-y"]'^) to 0, which is very much conditioned by the 
observed spectrum for k. 
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5.2 Experiments on MNIST handwritten digits 

We use the Experimental setting of jKJ03l |. also used in [CFV05| to compare 
the three previous kernels, namely, we sample 1.000 images from the MNIST 
database, that is 100 images per digit, and sample randomly clouds-of-pixels 
to compare such digits using the three kernels described above. The images, 
which are actually 28 x 28 matrices, are considered as clouds-of-pixels in the 
[0, 1]^ square, and we use a Gaussian kernel of width a — 0.1 to evaluate the 
similarity between two pixels through k, and use a three fold cross validation 
with five repeats to evaluate the performances of the kernels. The preliminary 
results shown in Table [1] show that the kernel V'Af is competitive with both 
and the inverse generalized variance, which was itself shown to be effective with 
respect to other kernels in [CFVOSt . such as simple polynomial and Gaussian 
kernels. 



Sample Size 


^Ao,?? = 0.01 




tpM,5 = 1 


40 pixels 


16.2 


28.6 


20.62 


50 " 


14.7 


16.47 


15.84 


60 " 


14.5 


14.97 


13.52 


70 " 


13.1 


11.3 


13 


80 " 


12.8 


10.8 


12.4 



Table 1: Misclassification rate expressed in percents for the 3 s.s.p.d. functions 
used on a benchmark test of recognizing digits images, with 40 to 80 black points 
sampled from the original images. 
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